CHX Loses A Single Data Center’s Database Access AND Production FileShare, But Network Between Data Centers is OK.

Situation

Impacts

Response

Database Access is lost in either CH2 or NY4, but network connectivity between the two data centers is in tact.

 

Production FileShare access in affected data center is also lost.

 

During Trading Hours:

 

The same exact impacts as when a single data center’s database access is lost with a working production fileshare, with the exception that the following local environment variables will need to be modified in order to support application restarts:

 

CHX_APP_CONFIG

IPC_CONFIG

IPC_CONFIG_LOC

TNS_ADMIN

 

All of these will need to change from the generic \\chx.com\prod\chxappcfg value to either a CH2 or NY4 specific production fileshare value.

 

 

1)      Redirect the affected data center’s servers to the working data center’s production fileshare.  These are the steps required to do this:

-          If Altiris is available, Tech Services has jobs to redefine local environment variables as necessary. 

-          If Altiris is not available, therer are registry key files in \\keymaster that must be manually imported on every server.  To import registry key files on a server:

A)      Login to Server

B)      Via Windows Explorer, find registry key file in \\keymaster\it_operations\controlroom\Test Reg Keys

C)      Double click the registry key file and follow prompts to import the keys.

2)      Once local environment variables have been changed, Tech Services will need to reboot all affected servers. 

-          If Altiris is available, Tech Services can use it.

-          If Altiris is not available, each server must be logged onto and restarted manually, individually.

3)      Follow the same exact procedure used when a single data center’s database access is lost with a working production fileshare.

 

Scope of Impact Response Considerations:

1)      This scenario documents a very specific problem with a well defined scope of impact and response.